Wednesday, October 27, 2004

Fermion Sign Problem Officially Hard

It now official: the fermion sign problem is hard. Matthias Troyer and Uwe-Jens Wiese present a proof that the fermion sign problem is NP-hard. They do this by mapping the quantum problem to a spin glass model, which is known to be NP-complete.


Do I believe it?


Hmmm. The results presented deal with a discrete problem. I'm not sure if the results apply to continuous systems as well or not. The whole notion of NP-completeness and combinatorial complexity deals with discrete systems. For continuous systems, information-based complexity (IBC) might be a better way of analyzing the problem. Note that the "fruit-fly" of IBC research is multidimensional integration.


However, the distinction between continuous and discrete might not be important. For instance, with global optimization, finding the locations of the local minima is "easy", but then one is left picking between many different discrete local minima to find the global minima (which makes it a potentially "hard" combinatorial problem - assuming the number of local minima grows exponentially with problem size).


I'm still perplexed as the origins of the difficulty of the FSP (actually, I don't understand the origin of difficulties for most NP problems.) So a quick look into the form of the integrals is in order.


We start with wanting the integral I = int f(x) dx, where x is really a vector of high dimension. Monte Carlo fits the bill nicely, having error bounds independent of dimension. But f(x) is strongly peaked - only small volumes contribute to most of the integral. To be efficient, we need importance sampling. The usually trick is to split f(x) into a product: p(x)g(x), where p(x) >= 0. Since p is positive, we can interpret it as a probability and use various sampling methods to get int p(x) g(x) dx / int p(x) dx.


We can get away with this in QMC by assuming that p(x) = psi^2(x), which is positive. But in Diffusion Monte Carlo, p(x) = psi(x) phi(x), where psi(x) is a guess and phi(x) is the exact ground state wavefunction. Now psi and phi will not have zeros at exactly the same location, so p(x) will have some negative regions.


The trick for dealing with negative p(x) is to split it once again into: sgn(p) abs(p) and sample abs(p), and move sgn(p) over with g(x). The integral now looks like
[int abs(p) sgn(p) g(x) / int abs(p)]/[int abs(p) sgn (p) / int abs(p)]


Here's where I get lost. One source of the problem could the denominator - since it is the average of a signed quantity, it could be small (at least relative to the errors) and magnify errors in the numerator? But the paper says that both the numerator and denominator have errors that grow exponentially in system size. So the denominator is an irritant, but not the root cause.


The error of the average sign (again from the paper) is related to the exponential of the free energy difference between the fermionic problem (with p(x)) and bosonic problem (with abs(p(x)). So the fermionic free energy is necessarily smaller because of cancellation induced by minus signs. But I still don't see how this makes the problem hard.


Why not add a sufficiently large constant to p to make it postive? Now there are no minus signs anywhere. But the essential features of the integrand haven't changed - so it can't be a "sign" problem. I guess I view the complexity of integration as coming from the unpredictable fluctations of the integrand - wildy fluctating integrands are "harder" to do than smooth integrands. Adding a constant doesn't change this.


Perhaps the antisymmetry requirements produce a function that has nearly equal positive and negative parts. Each one separately has a small relative error, but the nearly exact cancellation causes great magnification of the error (a well known numerical analysis bugaboo). Thus, the experiment of adding a constant doesn't change anything - it's added equally to both parts, and cancels exactly.

Lennard-Jones Server, Part II

Some more thoughts, in a Frequently Asked Questions (FAQ) format. (Actually, no one has asked any questions, so maybe it should be Frequently Unasked Questions. Or maybe not, given the acronym. Perhaps I should call them Frequently Anticipated Questions.)


Q. Why remote computation - why not just download a program to the users computer?
A. Have you ever downloaded software and tried to obtain all the dependencies? Hopefully, the client program would be much simpler. Other ideals are bug fixes and enhancements would take place on the server and automatically be picked up by the users (although this would require versioning to meet reproducibility requirements for researchers). The general idea of web services is somewhat of a holy grail for business types - getting paid every time someone uses the service, or getting paid on a subscription basis. Which is why word processing software is unlikely to take off as a web service any time soon. Getting someone to pay for a Lennard-Jones server is also unlikely.


Q. Why not just have a web page? Like Google, Amazon, etc.
A. A web service allows interfacing to these services under program control, which allows your computer to buy stuff on Amazon when it gets bored.


Q. Why a single component Lennard-Jones system? It's not exactly a hotbed of research.
A. No, but it is a testbed. New techniques are often tested on simple systems, and benchmark results would be useful. More importantly, it is a simple first test of the simulations over web services idea. I can see expanding to more complicated classical systems and QMC later.


Q. Can you explain more about accumulating a database of properties?
A. Yes. Aggregating data is a noble and useful goal, but it requires careful vetting of the input data. The idea behind a Lennard-Jones server is that users would only input physical parameters, not raw data. The data aggregation could then be automated, since the data (ie, the output of the program) should be of good quality (or at least of known quality).


The particular example I have in mind is computing free energies and the phase diagram. Computing a grid of pressures or energies at a number of state points is relatively easy, but computing free energies and such is harder - it can require integrating over a number of state points (if you use thermodynamic integration). Hopeful as more state points get put into the system, the free energies get more accurate. Calculations done during the idle time could be planned specifically to lower the errors on edges of the phases.


Now such a database will likely be inferior to targetted research trying to answer a particular question (ie, accurate freezing points, etc), but it seems like a good source of background information or a starting point. Also, the background or baseline information would be in electronic form, and as such be more useful than in a paper. (Not that papers aren't important - the raw data needs to be intrepreted and understood)


Q. Who would be likely users for such a service?
A. On the computation side, companies and researchers that use the results of simulations, but don't care about the details of the simulations. For researchers in the particular field the web service covers, it might not be so useful (the whole "not outsourcing your core competency" thing), although it would be useful for benchmarking and comparison of results.


On the data side, having a good collection of properties for the desired system would be useful for everyone.


Q. What would you do if it were the late 90's?
A. I would run out and register eSimulations.net.com and/or eWavefunctionsRUs.com, write a buzzword compliant business plan about the wonderful opportunies in B2C, B2B, R2B and R2R markets (the last two, made up on the spot, are Reseacher to Business and Researcher to Researcher.), and how actual experiments in the real world are passe, and the new e-world of the internet will make the real world obsolete. And try to get lots of VC money and go public with an overhyped IPO. Ah. For the good old days.

Sunday, October 24, 2004

Comments enabled

I've enabled comments. Just for registered users for now.
And I've started adding titles to the posts. Enjoy

Saturday, October 23, 2004

Lennard-Jones Server



If I could write blog entries as funny as Rory Blyth, I'd be famous. Well, maybe not. I'd have to draw cartoons, too. But to really be famous, I'd have to gratutiously link to Robert Scoble. (And get him to link back to this blog, the really hard part.) (For those who have non-computer-geek lives, Scoble wrote an article on how to get your blog noticed.)


The vaguely relevant part of that last paragraph is Rory's comments about XML Dev Con 2004. Some of the talks related to Web Services.


Web Services - what are they? Zef Hemel has a good description of Service Oriented Architecture, which basically means using web services (ie, Remote Procedure Calls over port 80) to construct an application.


That leads me to the Grid, of which web services seem to be a part. (But don't get me started on the popular analogy between the power grid and the computational grid. It's seductive and appealing, and perhaps even useful at a very abstract level, but the details break down badly if you push it very far (see the Scientific American article, among others.)


The grid concept seems to encompass four different things (as far as I can tell)


  1. Connecting geographically (and/or institutionally) separated supercomputers to work on a single calculation.
  2. Harvesting idle cycles on desktop PC's
  3. Running specific computations on remote machines (this is web services)
  4. Running arbitrary computations on remote machines (like the recent Sun announcement, but other companies have offered similar services previously)


Concerning items 3 and 4, I suspect web services will be more popular then running arbitrary computations remotely. If you have enough computational or data requirements to entertain thoughts of rent-a-FLOP (or rent-a-MIP), you will care about the environment and how well the code is optimized for the target architecture (or at least how predictable the run time is). But caring about such things makes buying "generic" computing power harder, or at least more complex.


But web services are (?) about hiding those nasty details. The provider can optimize the code for the target architecture or other details, and the user need not worry. It may be useful to divide the services into two kinds, one that requires lots of data, and the other that requires lots of compuations. For data-heavy services, the provider has lots of specialized data that is updated frequently (think Google, Amazon, Ebay, or stock market info), and delivering the data + program to the user would impractical. (versus something like TurboTax, which contains lots of specialized knowledge that changes yearly, but is practical to deliver the data + program to the user's computer. Although it can be and is offered over the web as well). For computation heavy services, optimization for particular hardware, etc could be considered specialized knowledge as well (making the divisons less distinct), but I still think the division may be useful.


As a way of understanding web services, and grid concepts, I was thinking of a Lennard-Jones server. A user could request properties of a single component system interacting via a Lennard-Jones potential for a given temperture/density/system size, the server would run the simulation and return the energy,pressure, g(r), and whatever else. (It would require limits on the system size to keep from completely overloading the server).


The server could have other features, such a keeping a database of run results, and returning the simulation results nearest the requrest parameters. Although I'm not sure this would work so well if one extended the idea to more complicated systems with larger parameter space.


In some ways, it's like a database of material properties, except that the properties are all generated by computation. So the database could be improved by further computation - if the server is idle it could work on reducing errors in various quantities. Although this might wreak havoc on reproduceability for users - good versioning would be needed.


Saturday, October 16, 2004

3D Function Plotter



In the continuing quest to visualize wavefunctions, I've put up the first version of a 3D function plotter.